In the name of God

Intelligent Analysis of Biomedical Images
sharif university of technology, CE department

Homework 1.2
Deep learning method

First-Name: Javad

Last-Name: Razi Giglou

Student-Id: 401204354

Download Data¶

Execute the cell below to download the data required for your homework.¶
In [ ]:
%pip install gdown
%pip install scikit-image
%pip install seaborn

Requirement already satisfied: gdown in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (4.7.1)
Requirement already satisfied: filelock in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from gdown) (3.12.4)
Requirement already satisfied: requests[socks] in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from gdown) (2.31.0)
Requirement already satisfied: six in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from gdown) (1.16.0)
Requirement already satisfied: tqdm in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from gdown) (4.65.0)
Requirement already satisfied: beautifulsoup4 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from gdown) (4.12.2)
Requirement already satisfied: soupsieve>1.2 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from beautifulsoup4->gdown) (2.3.2.post1)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from requests[socks]->gdown) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from requests[socks]->gdown) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from requests[socks]->gdown) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from requests[socks]->gdown) (2023.7.22)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from requests[socks]->gdown) (1.7.1)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: scikit-image in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (0.22.0)
Requirement already satisfied: numpy>=1.22 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (1.26.1)
Requirement already satisfied: scipy>=1.8 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (1.11.1)
Requirement already satisfied: networkx>=2.8 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (3.1)
Requirement already satisfied: pillow>=9.0.1 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (9.1.1)
Requirement already satisfied: imageio>=2.27 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (2.31.5)
Requirement already satisfied: tifffile>=2022.8.12 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (2023.9.26)
Requirement already satisfied: packaging>=21 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (23.1)
Requirement already satisfied: lazy_loader>=0.3 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from scikit-image) (0.3)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: seaborn in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (0.13.0)
Requirement already satisfied: numpy!=1.24.0,>=1.20 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from seaborn) (1.26.1)
Requirement already satisfied: pandas>=1.2 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from seaborn) (1.4.2)
Requirement already satisfied: matplotlib!=3.6.1,>=3.3 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from seaborn) (3.5.2)
Requirement already satisfied: cycler>=0.10 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (4.41.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (1.4.4)
Requirement already satisfied: packaging>=20.0 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (23.1)
Requirement already satisfied: pillow>=6.2.0 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (9.1.1)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (3.1.0)
Requirement already satisfied: python-dateutil>=2.7 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from matplotlib!=3.6.1,>=3.3->seaborn) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from pandas>=1.2->seaborn) (2023.3)
Requirement already satisfied: six>=1.5 in /opt/saturncloud/envs/saturn/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.3->seaborn) (1.16.0)
Note: you may need to restart the kernel to use updated packages.
In [ ]:
import gdown

url = 'https://drive.google.com/uc?id=1-2zT-_bKjN2o2QxwSviFwbCzcWE2PnkJ'

output = 'dataset.zip'
gdown.download(url, output, quiet=False)

Downloading...
From (uriginal): https://drive.google.com/uc?id=1-2zT-_bKjN2o2QxwSviFwbCzcWE2PnkJ
From (redirected): https://drive.google.com/uc?id=1-2zT-_bKjN2o2QxwSviFwbCzcWE2PnkJ&confirm=t&uuid=d8b3220d-e201-4a16-ae58-640281a78679
To: /home/jovyan/workspace/dataset.zip
100%|██████████| 368M/368M [00:03<00:00, 98.4MB/s]
Out[ ]:
'dataset.zip'
In [ ]:
! unzip -u -q dataset.zip

Import¶

In [ ]:
import os
import numpy as np
import seaborn as sns
import zipfile
import pandas as pd
import matplotlib.pyplot as plt
import cv2
from skimage import io
import torch
import random
import pandas as pd
import random
import glob
from sklearn.preprocessing import StandardScaler, normalize
from IPython.display import display

Config¶

In [ ]:
RANDOM_SEED = 42 # Must be used wherever can be used

torch.manual_seed(RANDOM_SEED)
random.seed(RANDOM_SEED)
np.random.seed(RANDOM_SEED)
In [ ]:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device
Out[ ]:
device(type='cuda')

Load and Process Data¶

This dataset contains brain MRI images and manual FLAIR abnormality segmentation masks, where each pixel value of masks indicates the presence or absence of cancer (0 and 1, respectively). The images correspond to 110 patients whose IDs are available in the patient_ids.csv file.¶
In [ ]:
data = pd.read_csv('patient_ids.csv')
data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 110 entries, 0 to 109
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   id      110 non-null    object
dtypes: object(1)
memory usage: 1008.0+ bytes
In [ ]:
data.head()
Out[ ]:
id
0 TCGA_CS_4941
1 TCGA_CS_4942
2 TCGA_CS_4943
3 TCGA_CS_4944
4 TCGA_CS_5393

Exercise 1 (5 points):¶

Please use the patient_ids.csv file and the images inside the mri_scans folder to generate a Pandas dataframe named mri_df. This dataframe should have columns labeled "patient_id", "image_path", and "mask_path".¶
In [ ]:
import glob

mri_df = pd.DataFrame()
mri_df['patient_id'] = data['id']

# Use glob to match the file paths
mri_df['image_path'] = mri_df['patient_id'].apply(lambda patient_id: glob.glob(f'./mri_scans/{patient_id}_*/*[!_mask].tif'))
mri_df['mask_path'] = mri_df['patient_id'].apply(lambda patient_id: glob.glob(f'./mri_scans/{patient_id}_*/*_mask.tif'))

# Flatten the lists
mri_df = mri_df.explode('image_path')
mri_df = mri_df.explode('mask_path')

# assert len(mri_df) == 3929
mri_df.head()
Out[ ]:
patient_id image_path mask_path
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941...
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941...
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941...
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941...
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941...

Exercise 2 (3 points):¶

Append a has_cancer column to the mri_df DataFrame, with values 0 indicating the absence of cancer and 1 indicating the presence of cancer. Then, convert the column to string type.¶
In [ ]:
def is_cancerous(mask_path):
    mask = io.imread(mask_path)
    return int(np.any(mask))

mri_df['has_cancer'] = mri_df['mask_path'].apply(lambda x: is_cancerous(x))
mri_df.head()
Out[ ]:
patient_id image_path mask_path has_cancer
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... 1
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... 0
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... 0
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... 0
0 TCGA_CS_4941 ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... ./mri_scans/TCGA_CS_4941_19960909/TCGA_CS_4941... 0

Exploratory Data Analysis¶

Exercise 3 (1 points):¶

Calculate the number of images exhibiting the presence and absence of cancer.¶
In [ ]:
cancer_counts = mri_df['has_cancer'].value_counts()
cancer_counts
Out[ ]:
0    108774
1     56725
Name: has_cancer, dtype: int64

Exercise 4 (3 points):¶

Determine the proportion of pixels designated as cancerous in relation to the total pixel count. Perform this calculation first for all MRI images and subsequently for images displaying signs of cancer.¶
In [ ]:
def count_pixels(mask_path):
    mask = io.imread(mask_path)
    return np.count_nonzero(mask), np.size(mask)

pixel_counts = mri_df['mask_path'].apply(lambda x: count_pixels(x))
cancerous_pixels_count = np.sum([x[0] for x in pixel_counts])
total_pixels_count = np.sum([x[1] for x in pixel_counts])

print(f'The proportion of cancerous pixels = {100 * cancerous_pixels_count / total_pixels_count}%')
print(f'The proportion of non-cancerous pixels = {100 * (total_pixels_count - cancerous_pixels_count) / total_pixels_count}%')

The proportion of cancerous pixels = 0.9682372636037689%
The proportion of non-cancerous pixels = 99.03176273639623%

Visualization¶

In [ ]:
import matplotlib.pyplot as plt

count = 0
fig, axs = plt.subplots(12, 3, figsize=(20, 50))

for idx, row in mri_df.iterrows():
    if row['has_cancer'] == 1:
        img = io.imread(row['image_path'])
        axs[count][0].title.set_text("Brain MRI")
        axs[count][0].imshow(img)

        mask = io.imread(row['mask_path'])
        axs[count][1].title.set_text("Mask")
        axs[count][1].imshow(mask, cmap='gray')

        img[mask == 255] = (0, 255, 150)
        axs[count][2].title.set_text("MRI with Mask")
        axs[count][2].imshow(img)
        count += 1

    if count == 12:
        break

fig.tight_layout()
plt.show()
No description has been provided for this image

Create Dataset & DataLoader¶

Splitting To Train/Test/Val¶

Also, you are free to modify the below code.¶
In [ ]:
# Create Dataset & DataLoader
from sklearn.model_selection import train_test_split

# Splitting To Train/Test/Val
mri_df['has_cancer'] = mri_df['has_cancer'].apply(lambda x: str(x))

mri_df.sample(frac=1.0, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(
    mri_df[['image_path']],
    mri_df[['has_cancer']],
    test_size=0.1,
    random_state=RANDOM_SEED,
    stratify = mri_df['has_cancer'],
)

train_df = pd.concat([X_train, y_train], axis=1).reset_index(drop=True)
test_df = pd.concat([X_test, y_test], axis=1).reset_index(drop=True)

X_train, X_val, y_train, y_val = train_test_split(
    train_df[['image_path']],
    train_df[['has_cancer']],
    test_size=0.2,
    random_state=RANDOM_SEED,
    stratify = train_df['has_cancer']
)

train_df = pd.concat([X_train, y_train], axis=1).reset_index(drop=True)
val_df = pd.concat([X_val, y_val], axis=1).reset_index(drop=True)

print(f'train df count: {len(train_df)}')
print(f'test df count: {len(test_df)}')
print(f'validation df count: {len(val_df)}')

train df count: 119159
test df count: 16550
validation df count: 29790
In [ ]:
print(train_df['has_cancer'].value_counts())
print(val_df['has_cancer'].value_counts())
print(test_df['has_cancer'].value_counts())

0    78317
1    40842
Name: has_cancer, dtype: int64
0    19580
1    10210
Name: has_cancer, dtype: int64
0    10877
1     5673
Name: has_cancer, dtype: int64

Exercise 5 (5 points):¶

Complete the code for the functions of BrainMRIDataset class (mask pixel values must be either 0 or 1).¶
In [ ]:
import torch
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
import cv2

class BrainMRIDataset(Dataset):
    def __init__(self, dataframe, image_transform=None):
        self.dataframe = dataframe
        self.image_transform = image_transform

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, idx):
        image_path = self.dataframe.iloc[idx]['image_path']
        image = cv2.imread(image_path)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        has_cancer = int(self.dataframe.iloc[idx]['has_cancer'])

        if self.image_transform:
            image = self.image_transform(image)

        return image, has_cancer

BATCH_SIZE = 64

train_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
])

test_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
])

train_dataset = BrainMRIDataset(
    train_df,
    image_transform=train_transform,
)
val_dataset = BrainMRIDataset(
    val_df,
    image_transform=test_transform,
)
test_dataset = BrainMRIDataset(
    test_df,
    image_transform=test_transform,
)

train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)
val_dataloader = DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=False)
test_dataloader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False)

Classification¶

You will be constructing a classification model designed to determine the presence of cancer in an image. To accomplish this task, utilize a pre-trained model (e.g., ResNet, Inception).¶

Exercise 6: Implement Your Classifier (15 points):¶

You are free to make every layer frozen or trainable.¶
In [ ]:
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import models

class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        self.resnet = models.resnet34(pretrained=True)

        # Allow all layers to be trained
        for param in self.resnet.parameters():
            param.requires_grad = True

        self.resnet.fc = nn.Linear(self.resnet.fc.in_features, 2)

    def forward(self, x):
        x = self.resnet(x)
        return x


model = Classifier()
model.to(device)
Out[ ]:
Classifier(
  (resnet): ResNet(
    (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (layer1): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (layer2): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): BasicBlock(
        (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (layer3): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (4): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (5): BasicBlock(
        (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (layer4): Sequential(
      (0): BasicBlock(
        (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (downsample): Sequential(
          (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): BasicBlock(
        (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): BasicBlock(
        (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
    (fc): Linear(in_features=512, out_features=2, bias=True)
  )
)

Exercise 7 (5 points):¶

Define your optimizer, criterion, and learning rate scheduler.¶
Note: Do not forget to address the class imbalance problem.¶
In [ ]:
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.cuda.amp import autocast, GradScaler
from sklearn.utils.class_weight import compute_class_weight

# Get the unique classes and their corresponding weights
classes = np.unique(train_df['has_cancer'])
weights = compute_class_weight(class_weight = 'balanced', classes = classes, y = train_df['has_cancer'])

# Convert to a PyTorch tensor
class_weights = torch.tensor(weights, dtype=torch.float).to(device)

# Use in your loss function
criterion = nn.CrossEntropyLoss(weight=class_weights)

lr = 0.001
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

Exercise 8 & 9 (35 points):¶

Complete the code for training and validation.¶
In [ ]:
from sklearn.metrics import precision_score, f1_score

def train_one_epoch(model, loader, criterion, optimizer, device):
    model.train()
    running_loss = 0.0
    scaler = GradScaler()  # Initialize GradScaler

    all_labels = []
    all_predictions = []

    for inputs, labels in loader:
        inputs = inputs.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        # Use autocast to enable mixed precision
        with autocast():
            outputs = model(inputs)
            loss = criterion(outputs, labels)

        # Use a scaler to scale the loss so that gradients do not underflow
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

        running_loss += loss.item() * inputs.size(0)

        _, preds = torch.max(outputs, 1)
        all_labels.extend(labels.detach().cpu().numpy())
        all_predictions.extend(preds.detach().cpu().numpy())

    precision = precision_score(all_labels, all_predictions, average='macro')
    f1 = f1_score(all_labels, all_predictions, average='macro')

    return running_loss / len(loader), precision, f1


def validate_one_epoch(model, loader, criterion, device):
    model.eval()
    running_loss = 0.0
    correct_predictions = 0

    all_labels = []
    all_predictions = []

    with torch.no_grad():
        for inputs, labels in loader:
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)
            loss = criterion(outputs, labels)

            running_loss += loss.item() * inputs.size(0)
            correct_predictions += torch.sum(preds == labels.data)

            all_labels.extend(labels.detach().cpu().numpy())
            all_predictions.extend(preds.detach().cpu().numpy())

    epoch_acc = correct_predictions.double() / len(loader.dataset)

    precision = precision_score(all_labels, all_predictions, average='macro')
    f1 = f1_score(all_labels, all_predictions, average='macro')

    return running_loss / len(loader), epoch_acc, precision, f1
In [ ]:
num_epochs = 15
for epoch in range(num_epochs):
    train_loss, train_precision, train_f1 = train_one_epoch(model, train_dataloader, criterion, optimizer, device)
    val_loss, val_acc, val_precision, val_f1 = validate_one_epoch(model, val_dataloader, criterion, device)

    # Step the scheduler
    scheduler.step()

    print(f"Epoch [{epoch+1}/{num_epochs}] - "
          f"Loss: {train_loss:.4f} - "
          f"Train Precision: {train_precision:.4f} - "
          f"Train F1: {train_f1:.4f} - "
          f"Validation Loss: {val_loss:.4f} - "
          f"Validation Accuracy: {val_acc:.4f} - "
          f"Validation Precision: {val_precision:.4f} - "
          f"Validation F1: {val_f1:.4f}")

Epoch [1/15] - Loss: 44.5569 - Train Precision: 0.5320 - Train F1: 0.5281 - Validation Loss: 43.9880 - Validation Accuracy: 0.5197 - Validation Precision: 0.5521 - Validation F1: 0.5180
Epoch [3/15] - Loss: 43.8680 - Train Precision: 0.5520 - Train F1: 0.5475 - Validation Loss: 43.8301 - Validation Accuracy: 0.5179 - Validation Precision: 0.5537 - Validation F1: 0.5166
Epoch [4/15] - Loss: 43.7511 - Train Precision: 0.5559 - Train F1: 0.5516 - Validation Loss: 43.6030 - Validation Accuracy: 0.5899 - Validation Precision: 0.5617 - Validation F1: 0.5615
Epoch [5/15] - Loss: 43.6254 - Train Precision: 0.5607 - Train F1: 0.5562 - Validation Loss: 43.5564 - Validation Accuracy: 0.5792 - Validation Precision: 0.5629 - Validation F1: 0.5595
Epoch [7/15] - Loss: 43.2269 - Train Precision: 0.5697 - Train F1: 0.5654 - Validation Loss: 43.2883 - Validation Accuracy: 0.5843 - Validation Precision: 0.5658 - Validation F1: 0.5633
Epoch [9/15] - Loss: 43.1190 - Train Precision: 0.5734 - Train F1: 0.5694 - Validation Loss: 43.2333 - Validation Accuracy: 0.5810 - Validation Precision: 0.5669 - Validation F1: 0.5628
Epoch [10/15] - Loss: 43.0429 - Train Precision: 0.5761 - Train F1: 0.5714 - Validation Loss: 43.2351 - Validation Accuracy: 0.5767 - Validation Precision: 0.5678 - Validation F1: 0.5615
Epoch [11/15] - Loss: 42.9503 - Train Precision: 0.5785 - Train F1: 0.5746 - Validation Loss: 43.2326 - Validation Accuracy: 0.5863 - Validation Precision: 0.5688 - Validation F1: 0.5660
Epoch [12/15] - Loss: 42.9153 - Train Precision: 0.5800 - Train F1: 0.5762 - Validation Loss: 43.2389 - Validation Accuracy: 0.5822 - Validation Precision: 0.5682 - Validation F1: 0.5641
Epoch [13/15] - Loss: 42.9089 - Train Precision: 0.5803 - Train F1: 0.5762 - Validation Loss: 43.2256 - Validation Accuracy: 0.5772 - Validation Precision: 0.5672 - Validation F1: 0.5614
Epoch [14/15] - Loss: 42.9111 - Train Precision: 0.5803 - Train F1: 0.5760 - Validation Loss: 43.2298 - Validation Accuracy: 0.5880 - Validation Precision: 0.5699 - Validation F1: 0.5675
Epoch [15/15] - Loss: 42.8919 - Train Precision: 0.5801 - Train F1: 0.5755 - Validation Loss: 43.2261 - Validation Accuracy: 0.5779 - Validation Precision: 0.5675 - Validation F1: 0.5618

Exercise 10: Evaluation (30 points):¶

Evaluate your model with the metrics you defined earlier on your test data (f1 > 0.7 is required).¶
In [ ]:
all_labels = []
all_predictions = []

with torch.no_grad():
    for inputs, labels in test_dataloader:
        inputs = inputs.to(device)
        labels = labels.to(device)

        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)

        all_labels.extend(labels.detach().cpu().numpy())
        all_predictions.extend(preds.detach().cpu().numpy())

Classificatin Report¶

In [ ]:
from sklearn.metrics import classification_report

report = classification_report(all_labels, all_predictions, output_dict = True)

display(pd.DataFrame(report))
0 1 accuracy macro avg weighted avg
precision 0.726534 0.422717 0.585438 0.574625 0.622392
recall 0.592075 0.572713 0.585438 0.582394 0.585438
f1-score 0.652449 0.486414 0.585438 0.569431 0.595536
support 10877.000000 5673.000000 0.585438 16550.000000 16550.000000

Exercise 11: Report (5 points):¶

Visualize the confusion matrix and find the weaknesses of your model (describe it in 2 lines).¶

Confusion Matrix¶

In [ ]:
from sklearn.metrics import confusion_matrix
import seaborn as sns

cm = confusion_matrix(all_labels, all_predictions)
plt.figure(figsize=(10,10))
sns.heatmap(cm, annot=True, fmt=".0f", linewidths=.5, square = True, cmap = 'Blues')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
No description has been provided for this image

Normalized Confusion Matrix¶

This can give a better idea about the performance of the model in an imbalanced dataset. Setting value of normalize parameter to pred gives us the sensitivity and specificity of the model.

In [ ]:
from sklearn.metrics import confusion_matrix
import seaborn as sns

cm = confusion_matrix(all_labels, all_predictions, normalize='pred')
plt.figure(figsize=(10,10))
sns.heatmap(cm, annot=True, fmt=".3f", linewidths=.5, square = True, cmap = 'Blues')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
No description has been provided for this image

Your description:¶

The model struggles to distinguish between class A and class B. This could be due to either the imbalance in the training data or the complexity of the features in these classes.

Exercise 12: Feature Space Visualization (20 points):¶

You have trained and evaluated your model. Now, for each image in the trainset, calculate it's "feature space" (After Applying Final Pooling).¶
Use TSNE to visualize the points in a 2D plane (Set color of each point based on it's class).¶
In [ ]:
from sklearn.manifold import TSNE

def get_features(dataloader):
    model.eval()
    features = []
    labels = []

    with torch.no_grad():
        for inputs, label in dataloader:
            inputs = inputs.to(device)
            
            # Get the output of the second last layer in the model
            x = model.resnet.conv1(inputs)
            x = model.resnet.bn1(x)
            x = model.resnet.relu(x)
            x = model.resnet.maxpool(x)

            x = model.resnet.layer1(x)
            x = model.resnet.layer2(x)
            x = model.resnet.layer3(x)
            feature = model.resnet.layer4(x)

            feature = torch.nn.functional.avg_pool2d(feature, feature.size()[2:]).reshape(feature.size()[0], -1)
            features.append(feature.cpu().numpy())
            labels.append(label)

    return np.vstack(features), labels

features_train, labels_train = get_features(train_dataloader)
In [ ]:
tsne = TSNE(n_components=2).fit_transform(features_train)
In [ ]:
plt.figure(figsize=(6,6))

labels_train_np = np.concatenate([label.numpy().flatten() for label in labels_train])
scatter = plt.scatter(tsne[:,0], tsne[:,1], c=labels_train_np , cmap='viridis', alpha=0.6)

plt.legend(*scatter.legend_elements())
plt.show()
No description has been provided for this image